Toy system for asymmetric multiplayer game play

ABSTRACT

According to one aspect, a toy system is adapted for use in a multiplayer game, the toy system comprising a primary user device, a secondary user device, and one or more toys; wherein the primary and secondary user devices are operable in a common multiplayer game session; wherein the primary user device is configured to capture video images of a real-world scene including the one or more toys from a primary view point and to display the captured video images as a primary view of the real-world scene; wherein the secondary user device is configured to display a secondary view of a scene corresponding to the real-world scene as seen from a secondary view point; wherein the one or more toys comprise at least one user-manipulable object adapted to selectively be brought into a user-selected one of a plurality of predetermined states, at least a first one of the predetermined states having a first visual appearance; wherein the primary user device is configured to detect said first predetermined state in a primary view including the user-manipulable object, based on the first visual appearance; and wherein the secondary user device is configured to produce user-perceptible output, in response to the detected first predetermined state. Corresponding methods and computer systems for implementing the method are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of priority to Danish PatentApplication No. DK PA201970101, filed on 14 Feb. 2019. The content ofeach of the above referenced patent applications is incorporated hereinby reference in its entirety for any purpose whatsoever.

FIELD OF THE INVENTION

The invention relates to toy systems for combined multiplayer augmentedand virtual reality. The toy system may be a toy construction systemcomprising construction elements with coupling members for detachablyinterconnecting construction elements.

BACKGROUND

Toy systems and, in particular, toy construction systems have been knownfor decades. In particular, toy construction systems comprising modulartoy elements having coupling members for detachably interconnecting themodular toy elements with each other have gained high popularity. Thesimple building blocks have been supplemented with dedicated toyelements with a mechanical or electrical function to enhance the playvalue. Such functions include e.g. motors, switches and lamps.

More recently, toy systems that utilize augmented reality (AR) haveattracted increased interest. Augmented reality (AR) is a technologywhere a captured live view of one or more items of a physical,real-world environment is augmented by computer-generated content, suchas graphics, sound etc., i.e. where a user is presented with a compositerepresentation comprising the live view of the environment and thecomputer-generated content, e.g. in the form of an overlay ofcomputer-graphics onto the live view. An overlay may be additive to thenatural environment or it may mask or replace the live view of thenatural environment. For example, computer-generated graphics may berendered on top of the live view or as a replacement where parts of thelive view are replaced by computer-generated graphics. For the purposeof the present description, a computer-implemented system implementingAR will generally be referred to as an AR system. An AR system generallycomprises an image capturing device, a suitably programmed processingunit and a display.

In an AR system, image features are often detected and recognized withinthe captured view of a real-world scene, and the AR system may thengenerate a computer-generated image (or other computer-generatedcontent, such as sound, haptic feedback, etc.) in dependence upon thedesignated image feature and superimpose the generated image on thecaptured view (or otherwise render the computer-generated content).

It is generally desirable to engage multiple users in a game play toenhance the interactive experience and to enhance the entertainmentvalue.

U.S. Pat. No. 9,779,548 B2 (Jordan Kent Weisman) describes a multiuser,collaborative augmented reality (AR) game system, which employsindividual AR devices for viewing physical models that are recognizableto the camera and image processing module of the AR device. The systemis adapted to ensure that only uniquely recognizable models aresimultaneously active at a particular location. The system furtherpermits collaborative AR to span multiple sites, by associating a portalwith an anchor at each site. Using the location of their correspondingAR device as a proxy for their position, AR renditions of the otherparticipating users are provided. A disadvantage of the system accordingto U.S. Pat. No. 9,779,548 B2 is that it requires all participatingusers to have access to an individual AR device and to a physical model.

EP 2 886 171 A1 (Microsoft) is directed to a cross-platform, multiplayeronline experience that combines an augmented reality system with otherplatforms. For example, users of a see-through display device mayparticipate in an AR experience. Users of a computing device operatingon a different platform may participate in a cross-platformrepresentation of the AR experience. The cross-platform representationmay thereby bring the appearance of the AR experience to a user of acomputing device that is typically incapable of providing such an ARexperience. However, the disadvantage of such a cross-platform system isthat the stimulating and engaging aspects of a playful interaction withthe physical scene in an AR-enhanced physical play experience get lostin the transformation from a pure AR multiplayer experience into ahybrid system combining both AR and non-AR devices in a commonmultiplayer session.

It is thus desirable to provide a toy system for combined augmented andvirtual reality, which facilitates or improves the interactive nature ofsuch a hybrid environment that is adapted for the simultaneous operationof AR devices and non-AR-devices by multiple players in a commonsession.

Generally, it remains desirable to provide a close integration of thephysical toys of the toy system and a virtual reality generated on acomputer. It is further generally desirable to provide a toy system thatallows small children, e.g. pre-school children, to combine physicaltoys, e.g. one or more physical toy construction models, with a virtual,computer-generated play experience.

It is also generally desirable to provide a toy system that is suitablefor children without a detailed understanding of programming techniques,control systems, or the like.

It is also generally desirable to enhance the educational and play valueof toy systems.

At least some embodiments of the toy system and other aspects disclosedherein seek to address one or more of the above problems and/or otherproblems arising when applying augmented-reality to toy systems in ahybrid multiplayer environment combining both augmented reality andvirtual reality devices to participate and interact in a commonmultiplayer session.

SUMMARY

Disclosed herein are embodiments of a toy system adapted for use in amultiplayer game, the toy system comprising a primary user device, asecondary user device, and one or more toys. The primary and secondaryuser devices are operable in a common multiplayer game session. Theprimary user device is configured to capture video images of areal-world scene including the one or more toys from a primary viewpoint and to display the captured video images as a primary view of thereal-world scene. The secondary user device is configured to display asecondary view of a scene corresponding to the real-world scene as seenfrom a secondary view point. The one or more toys comprise at least oneuser-manipulable object adapted to selectively be brought into auser-selected one of a plurality of predetermined states. At least afirst one of the predetermined states has a first visual appearance. Theprimary user device is configured to detect said first predeterminedstate in a primary view including the user-manipulable object, based onthe first visual appearance, and the secondary user device is configuredto produce user-perceptible output, in response to the detected firstpredetermined state.

A primary user operating the primary user device may thus control thegame flow in a multiplayer session by modifying the state of theuser-manipulable object, capturing and detecting the new state (or thechange in state) of the user-manipulable object with the primary userdevice, and sharing that information in the multiplayer game sessionwith the secondary user device, where it is used to produce userperceptible output to a secondary user operating the secondary userdevice. In particular, the toy system thereby provides a platform for anasymmetric multiplayer game. The platform provides an additionalpossibility to the primary user for controlling the multiplayer session,which requires a physical interaction with a physical toy, such as a toymodel, comprising a user-manipulable object. The interaction issubsequently brought to life in a digital world, which can be sharedwith other players across different types of user devices. By sharingthe control information with the other players and causing auser-perceptible output in response, the toy system also allowssecondary players who do not have access to the model, or who do nothave an AR-device at their disposal, to join the game and have part inthe added functionality resulting from the recognizable user-manipulableobject added to the physical toys of the system, even though they cannotcontrol or directly detect it. It is also conceivable that the actualphysical manipulation of the state of a user-manipulable object may beperformed by any of the users, also by the secondary users, providedthey have physical access to the one or more toys comprising auser-manipulable object. This way the secondary user could e.g. trap aprimary user accidentally capturing the visual appearance of the objectthat has been manipulated by the secondary user. The accidentaldetection of such a trapping state may then be shared with themultiplayer session and cause a user-perceptible output at the secondaryuser device. For example, a primary user approaching a forbidden zone,which has been secured by the secondary user by setting such traps, mayaccidentally capture such a user-manipulable object set to a trappingstate and thereby cause an alarm to be triggered at the second userdevice, in response to the primary user “going into the trap”.

The primary user device comprises a processing unit, a user interfaceincluding a display, and an image capturing device. The primary userdevice is thus adapted for receiving input from a user and to provideuser-perceptible output to the user. The primary user device is furtherconfigured to capture a sequence of images of a real-world scene, thescene including the at least one user-manipulable object. The primaryuser device may further process the captured images to detect apredetermined state within at least one of the captured images, based ona recognized visual appearance of the user-manipulable object. Detectingthe predetermined state of the user-manipulable object may compriseproviding a captured image as an input to a computer vision process, inparticular a feature detection process or an object recognition process.Advantageously, according to some embodiments, the computer visionprocess is an object recognition process based on a neural network. Whena predetermined state is detected, the primary user device may provideinformation on the detected state of the user-manipulable object to amultiplayer session in a multiplayer game environment.

The secondary user device comprises a processing unit, a user interfaceincluding a display, and optionally an image capturing device. Thesecondary user device is thus adapted for receiving input from a userand to provide user-perceptible output to the user. The secondary userdevice is further configured to receive information on a state of auser-manipulable object from a multiplayer session in a multiplayer gameenvironment. When operated in a common multiplayer session with theprimary user device, the secondary user device is thus adapted toreceive information on the state of the user-manipulable object asdetected by the primary user-device. Information on the current state ofthe user-manipulable object, or information indicating a change in thestate of the user-manipulable object, may thus in a common multiplayersession be passed from the primary user device to the secondary userdevice. In response thereto, the secondary user device may then produceuser-perceptible output.

In some embodiments, the primary user device may further be configuredto generate and render computer-generated user-perceptible informationprompting a primary user of the primary user device to manipulate thereal-world scene by setting or changing a state of a user-manipulableobject present in the scene. Said computer-generated information mayinclude guidance of the primary user to point the capturing devicetowards a location of the user-manipulable object, so as to include saiduser-manipulable object in the primary view.

The toy system may further be configured to select and/or developsecondary instructions based on the detected state, or based on adetected change of state of the user-manipulable object. Furthermore,the toy system may be configured to execute the selected and/ordeveloped secondary programmed instructions. Furthermore, the secondaryprogrammed instructions may include generating user perceivable outputat the secondary device, and/or game code for execution at the secondaryuser device.

The toy system may also include multiple primary devices and/or multiplesecondary user devices, all being operable to join a common multiplayersession. A plurality of primary user devices and/or a plurality ofsecondary user devices may also be grouped into respective primary andsecondary teams playing against each other in a multiplayer team gamesession.

According to some embodiments, the predetermined states furthercomprises a second predetermined state, the second predetermined statehaving a second visual appearance different from the first visualappearance. The primary user device may be configured to detect saidsecond predetermined state in a primary view including theuser-manipulable object, based on the second visual appearance. Like thefirst predetermined state, also the second predetermined state may beuniquely recognizable. The primary user device may further be configuredto analyze the captured video images with respect to the second visualappearance of the user-manipulable object in order to detect said secondpredetermined state as a current state of the user-manipulable object.The primary user device may further be configured to provide informationindicative of the detected second predetermined state to the at leastone secondary user device, typically via the common multiplayer session.According to some embodiments, the secondary user device is configuredto produce user-perceptible output, in response to the detected secondpredetermined state.

According to some embodiments, the user-manipulable object has in eachone of the predetermined user-selectable states a respective uniquevisual appearance that is uniquely recognizable by the primary userdevice, when the user-manipulable object is within the primary view.

According to some embodiments, the primary user device is configured todetect a current state of the user-manipulable object in a primary viewincluding the user-manipulable object, based on the visual appearance ofthe user-manipulable object. Furthermore, the secondary user device maybe configured to produce user-perceptible output, in response to thedetected current state.

According to some embodiments, the primary user device is configured todetect a change in state of the user-manipulable object in a primaryview including the user-manipulable object, based on the visualappearance of the user-manipulable object. The primary user device mayfurther be configured to analyze or monitor the captured video imageswith respect to changes in the visual appearance of the user-manipulableobject so as to detect a change in state of the user-manipulable object.The primary user device may further be configured to provide informationindicative of the detected change to the secondary processing device. Achange in state may e.g. be detected by comparing the current state witha previous state of the user-manipulable object, e.g. a previouslydetected or otherwise initialized state stored as the previous state ofthe user-manipulable object in the multigame session, and determine thata change in state has occurred if a discrepancy is observed.Furthermore, the secondary processing device may be configured toproduce user-perceptible output, in response to the detected change.

According to some embodiments, the primary user device is anaugmented-reality (AR) device configured to display the primary viewaugmented with computer-generated content. Further according to someembodiments, the secondary user device is a virtual-reality (VR) deviceconfigured to display the secondary view as a computer-generated view ofa virtual scene corresponding to the real world scene. In someembodiments, the secondary user device may further be configured todisplay the secondary view augmented with computer-generated content, inresponse to the detected predetermined state. The computer-generatedcontent may be visual content, audible content, tactile content, or anycombination thereof. Advantageously in some embodiments, the primaryuser device is one of a smartphone, a tablet computer, a gamecontroller, and a head-mountable display. Advantageously in someembodiments, the secondary user device is one of a desktop computer, alaptop computer, a tablet computer, a smartphone, a game controller, anda head-mounted display.

According to some embodiments, at least one of the one or more toys isone of a number of pre-determined toy models and the primary user deviceis configured to identify said at least one toy model among thepre-determined toy models. Advantageously the secondary view at thesecondary user device is provided on the basis of said identification,preferably as a virtual representation of the identified toy model.Furthermore, the at least one user-manipulable object may be placed at apre-determined position on the toy model.

According to some embodiments, the user-manipulable object comprises afixed portion and a moveable portion adapted to selectively be broughtinto a plurality of predetermined positions with respect to the fixedportion, each user-selectable position corresponding to one of theplurality of predetermined states. The moveable part may beattached/joint to the fixed part through one of: a hinge; a swiveljoint; a ball joint; a rotational bearing; a sliding bearing; a guiderail, or any other suitable moveable link. According to someembodiments, the visual appearance of the user-manipulable objectincludes an indicator area with a predetermined visual characteristic.The predetermined visual characteristic may be one or more of: a color;a surface texture, a pattern, textual information; a combination ofalphanumeric symbols; a glyph, a one-dimensional bar code, atwo-dimensional bar code; and a shape.

According to some embodiments, the toy system comprises a processingmodule configured to provide a multiplayer game environment formultiplayer game session processing. According to some embodiments, theprimary and secondary user devices may be operable to communicatedirectly with each other in a common multiplayer game session. Accordingto some embodiments, the primary and secondary user devices are operableto communicate through a further data processing system in a commonmultiplayer game session. The further data processing system may be anetworked data processing system configured to provide networked accessto a multiplayer game environment for the primary and secondary userdevices. The further data processing system may be a remote and/ordistributed system, such as a cloud-based data processing system. Thefurther data processing system may be configured to host a multiplayergame session for the first and second user devices. The further dataprocessing system may store, and execute, game code. The further dataprocessing system may further store game related information, such as acollection with model-data on identifiable toys, e.g. for predefined toyconstruction models, and a collection with object-data onuser-manipulable objects, each being stored with correspondingstate-data on the predetermined states and the visual appearanceassociated with these predetermined states. The further data processingsystem may further store, and handle any user-related information, suchas access rights, user profile data, history, assets, game stats,information regarding toys associated with a user, in particularinformation regarding any toy models associated with the user andadapted for use in the toy system, and the like.

According to some embodiments, multiplayer game session processing mayoccur at the primary device, at the secondary device, and/or at afurther data processing system included in or cooperating with the toysystem to provide a multiplayer game environment, the third device maybe a local server or, preferably, may be a distributed/remote system,such as a cloud-based data processing system. The processing module mayhave stored information therein associated with the plurality ofpredetermined states of the user-manipulable object.

According to some embodiments, the one or more toys comprise a pluralityof modular toy elements, the modular toy elements comprising cooperatingcoupling elements for detachably coupling the modular toy elements toeach other. The one or more toys may comprise a predefined toyconstruction model constructed from a plurality of the modular toyelements. According to some embodiments, the primary device may beconfigured for identifying the toy construction model. Furthermore, theuser-manipulable object may be attached to a toy construction modelconstructed from a plurality of toy construction elements.

Further according to some embodiments, the user-manipulable object maycomprise coupling elements for detachably connecting theuser-manipulable object with one or more of the modular toy constructionelements. Furthermore, the user-manipulable object may be a toyconstruction element in itself, with a functionality allowing a user tomanipulate the object for setting a pre-determined state. Theuser-manipulable object may also be constructed from a plurality of toyconstruction elements so as to build such functionality.

It may be noted that the gist of the invention described here-above inthe context of the toy system may also be carried out in other ways,such as a corresponding method implemented on a suitably programmedcomputer system, including any of the steps explicitly described orimplicitly disclosed herein. The gist of the invention may furthermorebe carried out by embodiments of a toy construction set comprising oneor more of the following: a computer-readable medium having storedthereon a computer program adapted to cause a computer to perform thesteps of embodiments of the method explicitly described or implicitlydisclosed herein, instructions for obtaining such a computer program,and/or the like.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1-3 each show a prior art toy construction element.

FIG. 4 shows schematically an embodiment of a toy system as disclosedherein.

FIGS. 5-7 show schematically further embodiments of a toy system asdisclosed herein.

FIGS. 8 a/b-10 a/b show different examples of user-manipulable objectsin two different states, respectively.

FIG. 11 shows an example of an asymmetric multiplayer digital gameexperience.

FIG. 12 illustrates an example of a game world including variousobstacles allowing a player to hide.

DETAILED DESCRIPTION

Various aspects and embodiments of toy construction systems disclosedherein will now be described with reference to modular toy elements inthe form of bricks. However, the invention may be applied to other formsof toy construction elements and other forms of toys.

FIG. 1 shows a modular toy element with coupling pegs on its top surfaceand a cavity extending into the brick from the bottom. The cavity has acentral tube, and coupling pegs on another brick can be received in thecavity in a frictional engagement as disclosed in U.S. Pat. No.3,005,282. FIGS. 2 and 3 show other such prior art modular toy elements.The modular toy elements shown in the remaining figures have this knowntype of coupling members in the form of cooperating pegs and cavities.However, other types of coupling members may also be used in addition toor in-stead of the pegs and cavities. The coupling pegs are arranged ina square planar grid, i.e. defining orthogonal directions along whichsequences of coupling pegs are arranged. The distance betweenneighboring coupling pegs is uniform and equal in both directions. Thisor similar arrangements of coupling members at coupling locationsdefining a regular planar grid allow the modular toy elements to beinterconnected in a discrete number of positions and orientationsrelative to each other, in particular at right angles with respect toeach other. The modular toy elements shown here, in FIGS. 1-3, are ofthe passive type, without additional functionality beyond mechanicalmodel building, such as electromagnetic, electronic, optical, or thelike. However, some embodiments of toy construction sets may alsoinclude active modular toy elements that are configured to perform userperceptual functions, e.g. emit light, sound, impart motion, etc.

FIG. 4 shows schematically an embodiment of a toy system 100 asdisclosed herein. The toy system 100 is adapted for use in an augmentedreality enhanced asymmetric multiplayer game. The toy system 100comprises a primary user device 110 for use by a primary user, asecondary user device 120 for use by a secondary user, and one or morephysical toys 131 placed in a real-world scene 130. The primary andsecondary user devices 110, 120 are operable to join in a commonmultiplayer game session 140. The primary user device 110 is configuredto capture video images of the real-world scene 130 including the one ormore toys 131 from a primary view point 111 and to display the capturedvideo images as a primary view 112 of the real-world scene 130. Thesecondary user device 120 is configured to display a secondary view 122of a scene corresponding to the real-world scene 130 as seen from asecondary view point, which in the example of FIG. 4 is the point ofview of a playable character 92 associated with the secondary user. Theone or more toys 131 comprise at least one user-manipulable object 150adapted to selectively be brought into a user-selected one of aplurality of predetermined states, i.e. at least two predeterminedstates 151, 152, 153, 154. In at least a first one 151 of thepredetermined states 151, 152, 153, 154 the user-manipulable object 150has a first visual appearance, whereby it is recognizable when theuser-manipulable object 150 is in the first state 151. The primary userdevice 110 is configured to analyze the captured video images withrespect to the first visual appearance of the user-manipulable object150 so as to detect said first predetermined state 151 as a currentuser-selected state of the user-manipulable object 150 in a primary view112 including the user-manipulable object 150, based on a recognition ofthe first visual appearance. Information indicative of the detectedfirst predetermined state 151 may then be provided to the secondary userdevice 120. The secondary user device 120 may then produceuser-perceptible output, in response to said information indicative ofthe detected first predetermined state 151. For example, the toy systemmay comprise a primary tablet computer 110 having stored thereon anaugmented reality (AR) App executing an AR-based digital gameapplication. The primary tablet computer 110 may comprise a display anda digital camera. The primary tablet computer may further comprise aprocessing unit executing the AR App and a storage medium for storinggame-related data. It will be appreciated that, instead of a tabletcomputer, the toy system may include another type of suitably programmeddata processing device or system as the primary user device 110,including a display, a processing unit, and an image capture device.Examples of suitable primary data processing systems include a personalcomputer, a desktop computer, a laptop computer, a handheld computer, agame console, a handheld entertainment device, a smart phone, or anyother suitably programmable computer.

The display is operatively coupled to (e.g. integrated into) the primarytablet computer 110, and operable to display, under the control of theprocessing unit of the primary tablet computer 110, a video image. Inthe example of FIG. 4, the display is a touch-sensitive display allowingthe primary tablet computer to receive user inputs such that the primaryuser may interact with the AR-based digital game executed on the primarytablet computer 110. It will be appreciated that the data processingsystem may comprise alternative or additional user interface devices forallowing the primary user to interact with the digital game, e.g. bygestures, eye tracking, etc.

The digital camera is a video camera operable to capture video images ofa real world scene 130. In the example of FIG. 4, the video camera isintegrated into the handheld primary tablet computer 110, i.e. theprimary user may move the tablet computer around so that the currentfield of view of the digital camera covers different parts of the realworld scene 130 from different viewpoints. The real world scene 130 maye.g. include a surface such as a desktop, a floor, or the like on whicha variety of objects 131 are positioned. The digital camera isoperatively coupled to (e.g. integrated into) the primary tabletcomputer 110 and operable to forward the captured video image to theprocessing unit of the primary tablet computer 110.

The digital camera captures video images of the scene 130 and theprimary tablet computer 110 displays the captured video images on thedisplay. In the example of FIG. 4, the real-world scene 130 comprisesthe toy 131 which is shown within the field of view of the camera. Inaddition to the toy 131, the scene 130 may comprise further objects,such as toys, other household objects, or the like.

The captured video images are displayed by the primary tablet computer110 on its display. Therefore, a user may move the toy 131 around and/orotherwise manipulate the toy 131 within the field of view of the digitalcamera and view live video images from the digital camera of the toy 131and at least of parts of the real world scene 130. Alternatively oradditionally, the primary user may change the position and/ororientation of the digital camera so as to capture images of a (e.g.stationary) toy 131 from different positions. Additionally, the computermay be operable to store the captured video images on a storage device,such as an internal or external memory, of the computer, and/or forwardthe captured video to another computer, e.g. via a computer network. Forexample, the computer may be operable to upload the captured videoimages to a website.

The primary tablet computer 110 is suitably programmed to execute anAR-enabled digital game, during which the computer performs imageprocessing on the captured video images so as to detect a pre-determinedstate, or change in state, of a user-manipulable object 150 within thecaptured video image. Responsive to the detected state or change instate, the computer may be programmed to generate a modified videoimage, e.g. a video image formed as the captured video image havingoverlaid to it a computer-generated image, e.g. a video image wherein atleast a part of the captured video image is replaced by acomputer-generated image. The computer 110 is operable to display themodified video image on the display. For the purpose of the presentdescription, a computer operable to implement AR functionalityoperatively connected to a video camera and a display will also bereferred to as an AR system. Image processing methods for detecting ARmarkers and for generating modified video images responsive to detectedobjects are known as such in the art (see e.g. Daniel Wagner and DieterSchmalstieg, “ARToolKitPlus for Pose Tracking on Mobile Devices”,Computer Vision Winter Workshop 2007, Michael Grabner, Helmut Grabner(eds.), St. Lambrecht, Austria, February 6-8, Graz TechnicalUniversity).

The toy system may further comprise a secondary tablet computer 120having stored thereon a VR App executing a VR-based digital gameapplication. The secondary tablet computer 120 may comprise a displayand a processing unit executing the VR App and a storage medium forstoring game-related data. It will be appreciated that, instead of atablet computer, the toy system may include another type of suitablyprogrammed data processing device or system as the secondary user device120, including a display, a processing unit, and an optional imagecapture device. Examples of suitable secondary data processing systemsinclude a personal computer, a desktop computer, a laptop computer, ahandheld computer, a game console, a handheld entertainment device, asmart phone, or any other suitably programmable computer.

The display is operatively coupled to (e.g. integrated into) thesecondary tablet computer 120, and operable to display, under thecontrol of the processing unit of the secondary tablet computer 120, avideo image. The video image may be computer-generated, or at leastinclude computer-generated content, on the basis of information on thereal-world scene 130 captured by the primary user device 110 to form asecondary view 121 on the secondary user device 120 of a scenecorresponding to said real world scene. More particular, thecomputer-generated content may be produced on the basis of informationon the one or more toys 131 in the real world scene 130. In the exampleof FIG. 4, the display is a touch-sensitive display allowing thesecondary tablet computer 120 to receive user inputs such that thesecondary user may interact with the VR-based digital game executed onthe secondary tablet computer 120. In the example of FIG. 4, the userinterface comprises user input regions 123 that may be used to receiveinput, such as game related input from the secondary user. Inparticular, the user inputs 123 may be used to control a secondary pointof view for producing the secondary view, thereby virtually movingaround in the scene corresponding to the real world scene, and/or forperforming game related interactions. It will be appreciated that thesecondary data processing system may comprise alternative or additionaluser interface devices for allowing the secondary user to interact withthe digital game, e.g. by gestures, eye tracking, etc.

The primary and secondary user devices 110, 120 also comprisecommunication means for linking to each other, and/or to furthercomponents forming part of the toy system, and/or even to externalcomputing resources, such as networked computing resources, whereby theprimary and secondary user devices 110, 120 are adapted to join andinteract in a common multiplayer game session 140 through links 41. 42.

The secondary user device may also comprise a digital camera. It is evenconceivable that the secondary user device may be an AR-enabled device,and may be operable in a similar manner as the primary device asdescribed above. The difference is, however, that the primary userdevice 110 captures images of the real world scene, detects apre-determined state or a change in state of the user-manipulable object150 in at least one of said captured images, as indicated by arrow 31,and provides this information to the multiplayer session 140, asindicated by arrow 41. The secondary user device 120, on the other hand,receives this information on the state, or change in state, of theuser-manipulable object 150 detected by the primary user device 110 fromthe multiplayer session 140, as indicated by arrow 42, and uses thisinformation to produce user-perceptible output for the secondary user,in response to the detected first predetermined state 151 (or change instate).

For example, events may be presented and visualized in the secondaryview on the display of the secondary user device 120, in response to anevent triggered by a state, or change in state, of a user-manipulableobject 150 on the physical toy 131 in the real world scene 130 asdetected by the primary user using the primary user device 110. Suchevents visualized on the secondary user device 120 may require directaction, such as tasks to be performed by the secondary user to handle agiven situation incurred by the state or change in state. A primary usermay thus physically set a state 151, 152. 153, 154 of theuser-manipulable object 150, detect the set state, provide the state tothe multiplayer session 140 through link 41, and thereby influence thecourse or conditions of the game for a secondary user acting in the samemultiplayer session 140 from a secondary user device 120, through link42.

The toy system further comprises a toy 131. In the example of FIG. 4 thetoy 131 is a toy construction model constructed from a plurality ofmodular toy elements, e.g. modular toy elements of the type described inconnection with FIGS. 1-3. It will be appreciated, however, that otherembodiments may employ other types of toys.

FIGS. 5-7 show schematically further embodiments of a toy system asdisclosed herein. By way of a few examples, FIGS. 5-7 illustrateembodiments of the invention may be implemented in numerous differentconfigurations to bring about an enhanced asymmetric multiplayer gameexperience.

In the embodiment of FIG. 5, a primary user device 110 and multiplesecondary user devices 120 a-c are operated to join a common multiplayersession 140 through respective links 41, and 42 a-c. The primary userdevice 110 is located at a real world scene 130. The primary user device110 may interact with the real-world scene 130, as indicated by arrow31, to capture images of the real world scene 130. The real-world scene130 includes one or more toys 131 with a user-manipulable object 150.The primary user device 110 may detect a predetermined state (or changein state) of the user-manipulable object 150. The primary user device110 shares information on the detected state (or change in state) withthe multiple secondary user devices 120 a-c also present in themultiplayer session 140. The information on the detected state (orchange in state) is then used to produce user-perceptible output at thesecondary user devices 120 a-c. As shown in FIG. 5 by way of exampleonly, the primary user device 110 and the multiple user devices 120 a-cmay all be smart phones or tablet computers. Furthermore, the primaryand secondary user devices 110, 120 a-c may communicate directly witheach other in a local network formed directly between the devices and/orthrough the use of a wired or wireless network infrastructure. While theprimary user device 110 requires access to the real world scene 130 tocapture images thereof, such local presence is not necessarily requiredfor the secondary user devices 120 a-c. Any of the secondaryuser-devices 120 a-c may therefore also be located at a remote locationwith respect to the real-world scene 130. The multiplayer 140 sessionmay be implemented as programmed instructions executed by one of theuser devices 110, 120 a-c participating in the multiplayer game, or by aseparate data processing system (not shown) serving the participatinguser devices 110, 120 a-c. The multiplayer 140 session may also beimplemented as programmed instructions executed in a distributed mannerby a combination of any of the user devices 110, 120 a-c participatingin the multiplayer game, and/or a separate data processing system.

In the embodiment of FIG. 6, multiple primary user devices 110 a-b andmultiple secondary user devices 120 a-c are operated to join a commonmultiplayer session 140 through respective links 41 a-b, 42 a-c. Theprimary user devices 110 a-b are all located at a real world scene 130.The primary user devices 110 a-b may interact with the real-world scene130, as indicated by arrows 31 a-b, to capture images of the real worldscene 130. The real-world scene 130 includes one or more toys 131 with auser-manipulable object 150. One or more of the primary user devices 110a-b may detect a predetermined state (or change in state) of theuser-manipulable object 150. The relevant primary user device 110 a-bshares the information on the detected state (or change in state) withthe multiple secondary user devices 120 a-c also present in themultiplayer session 140. The information on the detected state (orchange in state) is then used to produce user-perceptible output at thesecondary user devices 120 a-c. As shown in FIG. 6 by way of exampleonly, the primary user devices 110 a-b and the multiple user devices 120a-c may include smart phones, tablet computers, and conventional laptopor desktop computers. Furthermore, the primary and secondary userdevices 110 a-b, 120 a-c may communicate directly with each other in alocal network formed directly between the devices and/or through the useof a wired or wireless network infrastructure. While the primary userdevices 110 a-b require access to the real world scene 130 to captureimages thereof, such local presence is not necessarily required for thesecondary user devices 120 a-c. Any of the secondary user-devices 120a-c may therefore also be located at a remote location with respect tothe real-world scene 130. The multiplayer 140 session may be implementedas programmed instructions executed by one of the user devices 110 a-b,120 a-c participating in the multiplayer game, or by a separate dataprocessing system (not shown) serving the participating user devices 110a-b, 120 a-c. The multiplayer 140 session may also be implemented asprogrammed instructions executed in a distributed manner by acombination of any of the user devices 110 a-b, 120 a-c participating inthe multiplayer game, and/or a separate data processing system.

In the embodiment of FIG. 7, a primary user device 110 and multiplesecondary user devices 120 a-c are operated to join a common multiplayersession 140 through respective links 41, 42 a-c. The primary user device110 is located at a real world scene 130. The primary user device 110may interact with a real-world scene 130, as indicated by arrow 31, tocapture images of the real world scene 130. The real-world scene 130includes one or more toys 131 with a user-manipulable object 150. Theprimary user device 110 may detect a predetermined state (or change instate) of the user-manipulable object 150. The primary user device 110shares information on the detected state (or change in state) with themultiple secondary user devices 120 a-c also present in the multiplayersession 140. The information on the detected state (or change in state)is then used to produce user-perceptible output at the secondary userdevices 120 a-c. As shown in FIG. 7 by way of example only, the primaryuser device 110 and the multiple user devices 120 a-c may include smartphones, tablet computers, and conventional laptop or desktop computers.In the example shown in FIG. 7, the primary and secondary user devices110, 120 a-c communicate with each other through a networkedinfrastructure. While the primary user device 110 requires access to thereal world scene 130 to capture images thereof, such local presence isnot necessarily required for the secondary user devices 120 a-c. Any ofthe secondary user-devices 120 a-c may therefore also be located at aremote location with respect to the real-world scene 130. Furthermore inthe example shown in FIG. 7, the multiplayer 140 session may beimplemented as programmed instructions executed by a cloud processingsystem 9 serving the participating user devices 110 a, 120 a-c. Themultiplayer 140 session may also be implemented as programmedinstructions executed in a distributed manner by a combination of suchcloud computing resources 9 and any of the user devices 110 a-b, 120 a-cparticipating in the multiplayer game, and separate local dataprocessing systems (not shown).

FIGS. 8 a/b, 9 a/b, and 10 a/b, show different examples of usermanipulable objects 150 with various states. The user-manipulableobjects 150 are each shown in two different pre-determined states,wherein at least a first pre-determined state 151 is detectable by aprimary user device 110 as discussed elsewhere herein. In the examplesshown in FIGS. 8 a/b, 9 a/b, and 10 a/b, the different states of theuser-manipulable object are realized by different mechanical functions.

The user-manipulable objects shown in FIGS. 8 a/b, 9 a/b, and 10 a/bcomprise a moveable portion 155 and a fixed portion 156. The moveableportion 155 is adapted to selectively be brought into a plurality ofpredetermined positions with respect to the fixed portion 156, whereineach user-selectable position corresponds to one of the plurality ofpredetermined states. In FIGS. 8 a/b, 9 a/b, and 10 a/b two differentstates are illustrated for each of the user-manipulable objects 150.

The user-manipulable object 150 shown in FIG. 8 a/b has a moveableportion 155, which is attached to a fixed portion 156 by means of ahinge H allowing the moveable portion 155 to be folded over the fixedportion 156. As seen in FIG. 8 a, the fixed portion 156 carries atwo-dimensional code on an indicator area thereof. The two-dimensionalcode may be recognizable in the primary user device 110, e.g. by acomputer vision process executed on the primary user device 110. Thetwo-dimensional code characterizes the visual appearance of a firststate 151 of the user-manipulable object 150 as seen in FIG. 8 a. Thefirst state 151 may thus be detected when the primary user devicerecognizes the two-dimensional code. In a second state shown in FIG. 8b, the moveable portion 155 is folded over the fixed portion 156, therebyclosing the user-manipulable object like a clam-shell to conceal thetwo-dimensional code. In some embodiments of the toy system, the closedstate of FIG. 8b may not be detectable by the primary device 110. Theuser-manipulable object shown in FIGS. 8 a/b is thus particular usefule.g. for activating an event or behavior in the multiplayer game, whenthe moveable portion 155 is brought into the OPEN position with respectto the fixed portion 156, such that the recognizable two-dimensionalcode on the indicator area can be detected.

The user-manipulable object 150 shown in FIGS. 9 a/b has a moveableportion 155, which is attached to a fixed portion 156 by means of asliding joint S allowing the moveable portion 155 to slide over thefixed portion 156 into different positions, in the present example twodifferent positions shown in FIG. 9a and FIG. 9 b, respectively. Thefixed portion carries two different symbols, which each may berecognized, and distinguished from one another, by a primary user device110. When the moveable portion 155 is in the left position, as seen inFIG. 9a , a happy smiley indicative of a recognizable firstpredetermined state 151 is seen. When the moveable portion 155 is in theright position, as seen in FIG. 9b , a sad smiley indicative of arecognizable second predetermined state 152 is seen. Theuser-manipulable object 150 of FIG. 9 a/b is thus useful e.g. as acontrol for switching between two distinguishable game related events orbehaviors.

The user-manipulable object 150 shown in FIGS. 10 a/b has a moveableportion 155, which is attached to a fixed portion 156 by means of arotatable joint R allowing the moveable portion 155 to be rotated withrespect to the fixed portion 156 into different positions. The rotatablewheel 155 may have a plurality of indicator areas that can berecognized, and distinguished, by primary user device 110. At leastadjacent indicator areas should be marked in a different manner so as tobe able to distinguish neighboring indicator areas from each other. Therotatable wheel 155 of FIGS. 10 a/b has four indicator areas, eachmarked with a different color as illustrated by different hatchings. Agiven predetermined state may be recognized when the rotatable wheel 155is positioned such that a specific one of the indicator fields isaligned with the solid marker of the fixed portion 156. Thereby, fourdifferent distinguishable states 151, 152, 153, 154 can be realized, anddetected by means of primary user device 110. The user-manipulableobject 150 of FIG. 9 a/b is thus useful e.g. as a control for switchingbetween a plurality of distinguishable game related events or behaviorsby means of a simple rotation. The rotation may be performed by a userin a controlled manner to purposively set a specific state 151, 152,153, 154. However, it is also conceivable that a user may spin therotatable portion 155 in a more randomized manner to allow for an aspectof chance in the outcome of the predetermined state 151, 152, 153, 154set by the physical interaction with the user-manipulable object 150.

Setting or changing a state of a user-manipulable object may thusinclude e.g. one or more of the following: moving an element of theuser-manipulable object, re-positioning an element of theuser-manipulable object, changing the orientation of an element of theuser-manipulable object. Moving, re-positioning, or rotating an elementmay entail revealing or concealing an indicator element carrying avisual characteristic, which is recognizable by the primary user device.According to some embodiments, it is also conceivable that such moving,re-positioning, or rotating of an element may cause a change in theoverall shape of the user-manipulable object. It may also cause a changein the orientation or in the exact positioning of elements of theuser-manipulable object with respect to remaining elements of theuser-manipulable object, or with respect to a toy to which it isattached. The resulting shape, position and/or orientation may also berecognizable by the primary user device. The result of the recognitionof the visual appearance of the user-manipulable object may then be usedfor detecting a predetermined state of the user-manipulable object.Alternatively or in addition thereto a change in the recognizable visualappearance may be recognized and registered as a change in the state ofthe user-manipulable object.

While in FIGS. 8 a/b emphasis is put on mechanical functionality toprovide a user-manipulable object with different predetermined states,it is also conceivable to provide a user-manipulable object with aplurality of visually distinguishable predetermined states by any othersuitable means. For example, a visual appearance that may be recognizedin images captured by a primary user device may also be realized bymeans of electrically, electronically, or digitally controlled indicatormeans, such as one or more light emitters or a display.

Turning now to FIGS. 11 and 12, an example of an asymmetric digital gameexperience with one AR-enabled primary device (AR Player 1) and fourVR-enabled devices (Player 1-4) meeting in a common multiplayer sessionin a 3D game world is briefly described.

FIG. 11 shows schematically a primary user device 110 capturing a realworld scene 130 comprising a toy model 131 of e.g. a building, and auser-manipulable object 150. The real-world scene may be set on a tablein a room where a primary user operating the primary user device 110 maybe located. The primary user device may be operated to join amultiplayer session 140, here represented by a three-dimensional gameworld defining a virtual scene corresponding to the real world scene.The primary user device may show a primary view of the real world scene130 augmented with computer-generated content to provide anAR-experience. The primary view is shown as seen from a primary viewpoint, which may be controlled e.g. by moving the primary user device110 with respect to the real world scene 130, wherein thecomputer-generated AR-content is tracked and presented on a displayscreen of the primary user device 110 accordingly. In addition to acomputer-generated virtual representation 141 of the toy model 131, themultiplayer session 140 comprises further computer-generated virtualobjects 142 that may be thematically linked to the toy model 131 and itsvirtual representation 141. For example, if the toy model 131 is apirate ship, the virtual scene 140 may be set on a deserted islandsurrounded by dangerous reefs, and the additional virtual objects 142may include palm trees, primitive huts, and a treasure cave.Furthermore, FIG. 11 shows secondary user devices 120 a-d, which may ormay not be at the same location as the real world scene and the primaryuser device. The secondary user devices 120 a-d may be operated to jointhe same multiplayer session 140 as the primary user device 110. Thesecondary user devices 120 a-d may access the multiplayer session 140through a local or remote link depending on the location and the actualimplementation of the multiplayer session 140 as discussed elsewhereherein. The secondary user devices 120 a-d can then on a display thereofshow a respective secondary view 122 a-d of the virtualthree-dimensional game world including the virtual representation 141 ofthe toy model 131 and any of the additional virtual objects 142,according to the respective secondary view point of the secondary userdevice 120 a-d. The secondary view point may be the point of view of aplayable character associated with a given secondary user device 120a-d. The playable character may be controlled by the secondary userusing input controls 123, which may be touch sensitive input areas on agraphical user interface of the secondary user device 120 a-d, e.g.configured with buttons and/or a joy-stick control functionality.

In the above-mentioned pirate setting, the primary player (or a team ofprimary players not shown here) may play treasure hunter (or a group oftreasure hunters), and the secondary players may be left-behind piratestrying to defend the treasure. User-manipulable objects may then bemanipulated to modify the flow of the game e.g. changing the weather bya random spin of a color wheel, releasing wild animals, or for conjuringspirits of cursed pirates.

In another example, the toy model 131 may be a model of a haunted housefairground attraction, which then may be set in a virtual fairgroundscene equipped with additional objects 142, such as further fairgroundattractions like a merry-go-rounds, a roller coaster, a fortune teller'stent, or an ice cream and popcorn sales cart. The fairground may behaunted by ghosts played by secondary users using secondary user devices120 a-d. The ghosts may move around haunting different attractions ascontrolled by means of the user inputs 123 on the secondary user devices120 a-d. The ghosts may be of different kinds, each kind beingassociated with a pre-determined color. A primary user (or a team ofprimary players not shown here), may have the task to hunt the ghosts,but may only be able to see the ghosts of one color at a time, accordingto a current color setting. The current color setting may be determinedby means of a user-manipulable object 150, such as the above-mentionedcolor wheel, which may be controlled by the primary user. Moving aroundthe ghosts may also hide behind or under objects 141, 142 in the virtualscene 140. When a ghost comes into sight, the ghost hunter may catch oreliminate the ghost by aiming with the AR-enabled primary user device atthe virtual representation of the secondary player 92 a-d, here a ghostof a given color, and pressing at the FIRE button. Generally, the realworld scene may also comprise a plurality of user-manipulable objectsdistributed through the scene and with different functionalities andeffect on the game. For example, a user-manipulable object (not shown)may have a hatch concealing a symbol or code characterizing a detectablestate. By opening the hatch and detecting the revealed symbol or codewith the primary user device 110, a side-kick helping the ghost hunterto seek the hidden ghosts may be activated. At the secondary device achange in the current color setting may be indicated by a change incolor of an element of the avatar of the primary player, and may alsoactivate portals for the ghosts of the current color to escape intoanother color. In a similar manner, activating the side kick may resultin a visual, audible, tactile, and/or other user-perceptible output tobe produced on the secondary devices indicating the detected state (orchange in state) and/or modifying thereby the flow of the multiplayergame session.

Very briefly, FIG. 12 merely illustrates an example of a game worldincluding various obstacles allowing a secondary player to hide from aprimary player in e.g. a hide-and-seek type of game, such as theabove-mentioned specific implementation of a hide-and-seek concept in ahunter's game. The obstacles may include any of the computer-generatedobjects 141, 142 in the virtual scene 140.

As mentioned above, the different aspects of the present invention canbe implemented in different ways including the toy system described inrespect of the various embodiments of the aspects described above and inthe following. Further aspects of the present invention can e.g. beimplemented as corresponding methods, apparatus, and products. Eachaspect, yields one or more of the benefits and advantages described inconnection with at least one of the aspects described above, and eachaspect having one or more preferred embodiments corresponding to thepreferred embodiments described in connection with at least one of theaspects described above and in the following and/or disclosed in thedependant claims. Furthermore, it will be appreciated that embodimentsdescribed in connection with one of the aspects described herein mayequally be applied to the other aspects.

In particular, a computer-implemented method is provided for generatingan asymmetric multiplayer play experience, the process comprising:

-   -   capturing a sequence of images of a real world scene using a        primary user device, the real-world scene including at least one        toy with a user-manipulable object:    -   operating the primary user device and at least one secondary        user device to join in a common multiplayer digital game        session;    -   in the multiplayer game session, providing a virtual        representation corresponding to the real world scene;    -   processing the captured images to detect at least a first        predetermined state of the user-manipulable object within the        real-world scene;    -   providing information on the first predetermined state to the        multiplayer session:    -   responsive to the information on the detected predetermined        state, generating and rendering computer-generated        user-perceptible output associated with the detected        pre-determined state at the at least one secondary user device.

Furthermore, a data processing system is provided that is configured,e.g. by suitable program code stored on the data processing system, toperform the steps of one or more of the computer-implemented methodsexplicitly described or implicitly disclosed herein.

Yet further, a computer program product is provided comprising programcode adapted to cause, when executed on a data processing system, a dataprocessing system to perform the steps of one or more of thecomputer-implemented methods explicitly described or implicitlydisclosed herein.

The computer program product may be provided as a computer-readablemedium, such as a CD-ROM, DVD, optical disc, memory card, flash memory,magnetic storage device, floppy disk, hard disk, etc. In otherembodiments, a computer program product may be provided as adownloadable software package, an App, or the like, e.g. on a web serverfor download over the internet or other computer or communicationnetwork. In particular, an embodiment of a toy construction set mayinclude toy construction elements, and installation instructions forinstalling a computer program product on a suitable data processingsystem.

In the claims enumerating several means, several of these means can beembodied by one and the same element, component or item of hardware. Themere fact that certain measures are recited in mutually differentdependent claims or described in different embodiments does not indicatethat a combination of these measures cannot be used to advantage.

It should be emphasized that the term “comprises/comprising” when usedin this specification is taken to specify the presence of statedfeatures, elements, steps or components but does not preclude thepresence or addition of one or more other features, elements, steps,components or groups thereof.

1-29. (canceled)
 30. A method of controlling a multiplayer game systemcomprising the steps of: operating a primary user device and a secondaryuser device in a common multiplayer game session; capturing, with theprimary user device, a sequence of images of a real world sceneincluding a user manipulable object having a plurality ofuser-selectable states; modifying a user-selected state of theuser-manipulable object; detecting, with the primary user device, achange in the user-selected state of the user-manipulable object in thecaptured sequence of images; sharing information indicative of thedetected change in the user-selected state within the common multiplayergame session; providing secondary programmed instructions responsive tothe detected change in the user-selected state; and executing thesecondary programmed instructions at a secondary user device.
 31. Amethod as recited in claim 30, wherein the primary user device comprisesa processing unit, a user interface including a display, and an imagecapturing device, the primary user device being adapted to detect thechange in the user-selected state of the user-manipulable object byproviding the captured sequence of images to a computer vision processconfigured to recognize user-selected states of the user-manipulableobject, comparing a currently recognized user-selected state with astored previous user-selected state, and determining that a change hasoccurred if a discrepancy is observed.
 32. A method as recited in claim31, wherein the computer vision process includes a feature recognitionprocess and an object recognition process.
 33. A method as recited inclaim 30, wherein the primary user device is configured to generate andrender user-perceptible output prompting the primary user of the primaryuser device to manipulate the real world scene by setting a state of theuser-manipulable object.
 34. A method as recited in claim 33, whereinthe user-perceptible output includes: prompting the primary user to setthe user-manipulable object to a pre-determined user-selected state; andproviding guidance to point a capturing device of the primary userdevice towards a location of the user-manipulable object.
 35. A methodas recited in claim 30, wherein the secondary programmed instructionsinclude one or more of: game code; and instructions for generatinguser-perceptible output.
 36. A method as recited in claim 30, furthercomprising the step of grouping a plurality of primary and secondaryuser devices as teams in the common multiplayer game session.
 37. Amethod as recited in claim 30, wherein the primary user device and thesecondary user device are selected from the group consisting of:augmented reality enabled (AR) devices; and virtual reality enabled (VR)devices.
 38. A method as recited in claim 30, wherein the commonmultiplayer game session is implemented on components selected from thegroup consisting of the primary user device, the secondary user device,a networked processing device, and a cloud computing environment.
 39. Amethod of controlling a multiplayer game environment for multiplayergame session processing, the method comprising the steps of: receivingone or more primary user devices and one or more secondary user devicesin a common multiplayer game session; receiving from a primary one ofthe user devices a sequence of images of a real world scene including auser manipulable object, the user manipulable object having a pluralityof user-selectable states; detecting in the sequence of images a changein a user-selected state of the user-manipulable object; sharinginformation indicative of the change in the user-selected state withinthe common multiplayer game session; responsive to the change in theuser-selected state, providing secondary programmed instructions; andcausing the secondary programmed instructions to be executed at one ofthe one or more secondary user devices.
 40. A method as recited in claim39, wherein the one or more primary user devices and the one or moresecondary user devices are selected from the group consisting of:augmented reality enabled (AR) devices; and virtual reality enabled (VR)devices.
 41. A method as recited in claim 39, further comprising thestep of grouping a plurality of primary and secondary user devices asteams in the common multiplayer game session.
 42. A method ofcontrolling a multiplayer game environment for multiplayer game sessionprocessing, the method comprising the steps of: establishingcommunication between a plurality of user devices in a commonmultiplayer game session; receiving from a primary one of the userdevices a sequence of images of a real world scene including a usermanipulable object, the user manipulable object having a plurality ofuser-selectable states; detecting in the sequence of images a change ina user-selected state of the user-manipulable object; providingprogrammed instructions the plurality of user devices based on thechange in the user-selected state; and executing the secondaryprogrammed instructions.